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(DSN  664-9027)  (dw^e@dodig.osd.mil).  See  Appendix  F  for  the  report  distribution. 
The  team  members  are  listed  inside  the  back  cover. 
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Acquisition  Management  of  the  Army’s 
AU  Source  Analysis  System 


Executive  Summary 


Who  Should  Read  this  Report  and  Why?  DoD  acquisition  executives,  program 
managers,  and  independent  testers  and  evaluators  should  read  tiiis  rq)ort  because  similar 
management  control  issues  may  apply  to  other  weapons  and  information  technology 
acquisitions. 

Background.  The  All  Source  Analysis  System  (AS  AS)  is  an  Army  Acquisition 
Category  E,  mission-critical  weapon  system  that  has  been  in  development  since  1979.  It 
is  a  family  of  systems  being  acquired  and  deployed  by  evolutionary  blocks  and  spiral 
version  enhancements.  A  &y  ASAS  component  is  the  Remote  Workstation.  The 
combination  hardware  and  software  system  provides  automated  information  support  to 
intelligence  staff  elements  and  other  designated  intelligence  organizations. 

The  Program  Executive  Officer,  Command,  Control  and  Communications  Tactical  is  the 
ASAS  Milestone  Decision  Authority.  The  Project  Manager,  IntelUgence  and  Effects,  in 
coordination  with  the  Army  Training  and  Doctrine  Command  System  Manager  and  the 
Army  Test  and  Evaluation  Command,  a<^uires,  tests,  and  deploys  ASAS  hardware  and 
software  components.  Total  Army  acquisition  development  and  deployment  costs  for 
ASAS  could  exceed  $2.5  billion. 

Results.  The  Army  redesignated  the  ASAS  as  an  Acquisition  Category  E  program  when 
it  should  have  remained  an  Acquisition  Category  I  program.  Further,  toe  ASAS  Remote 
Workstation  with  Version  4  Software  did  not  demonstrate  intended  results  when  it  was 
tested  and  evaluated  by  toe  Army  Test  and  Evaluation  Command.  In  addition,  toe  Office 
of  toe  Director,  Operational  Test  and  Evaluation  e^^ressed  a  quahfied  opinion  in  its 
FY  1999  Annual  Report  that  toe  Remote  Workstation  should  be  fidded,  despite  problems 
with  software  interoperability  and  logistics  supportability. 

Management  controls  need  to  be  implemented  for  toe  ASAS  acquisition  to  ensure  that 
ASAS  products  demonstrate  results  in  accordance  with  operational  r^uirement 
documents.  Specifically,  toe  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  should  elevate  toe  acquisition  from  an  Acquisition  Category  E  Major 
Defense  Acquisition  Program  to  an  Acquisition  Category  I  Major  Defense  Acquisition 
Program  and  assume  responsibility  for  acquisition  management  oversight.  In  Edition, 
toe  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  in 
coordination  with  toe  Under  Secretary  of  Defense  (Intelligence),  should  establish 
benchmarks  for  critical  operational  issues  and  criteria  as  deployment  thresholds  for 
operational  tests.  Further,  the  Program  Executive  Officer,  Command,  Control  and 
Communications  Tactical  should  implement  comparative  cost,  schedule,  and  performance 
parameters  for  measuring  s^^tem  efficiency  and  effectiveness  and  for  projecting  program 
results;  and  test  and  evaliiation  results  should  validate  that  toe  ASAS  was  ready  for 


operational  testing  and  that  operators  and  analysts  were  sufficiently  trained  to  use  the 
system.  Also,  the  Director,  Operational  Test  and  Evaluation  should  provide  unqualified 
opinions  for  ASAS  effectiveness  and  suitability.  (See  the  Finding  section  of  the  report 
for  the  detailed  recommendations.) 

Management  Comments  and  Audit  Response.  The  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  nonconcurred,  although  he  a^eed  in  principle 
with  the  recommendation  to  elevate  the  ASAS  Program  to  an  Acquisition  Category  I  and 
provide  appropriate  acquisition  management  oversight  if  the  dollar  thresholds  for  a  major 
defense  acquisition  program  are  met.  He  will  decide  whether  to  evaluate  the  acquisition 
after  the  Major  Defense  Acquisition  Program  List  is  updated  in  Decanber  2003.  hi 
addition,  the  Under  Secretary  concurred  with  raising  the  dqiloyment  threshold  for 
operational  tests  and  will  issue  updated  program  guidance  if  the  ASAS  is  elevated  to  an 
Acquisition  Category  I  program. 

The  Program  Executive  Officer,  Command,  Control  and  Communications  Tactical 
concurred  with  the  recommendation  to  implement  comparative  cost,  schedule,  and 
performance  parameters.  The  Director,  G^erational  Test  and  Evaluation  concurred  in 
principle  with  providing  unqualified  opinions  on  the  effectiveness  and  suitability  of  tested 
systems.  However,  he  will  provide  his  opinion  on  the  ASAS  Block  II  after  the  initial 
operational  test  and  evaluation  rather  than  after  tests  of  system  components,  which  the  All 
Source  Analysis  System  Test  and  Evaluation  Master  Plan  requires.  Because  ttie  ASAS 
Block  n  development  is  near  completion,  we  request  that  the  Director,  Operational  Test 
and  Evaluation  provide  the  Inspector  General  of  the  Department  of  Defense  with  a  copy 
of  his  assessment  of  the  Army’s  ASAS  Block  II  Initial  Operational  Test  and  Evaluation 
after  tests  are  completed.  Also,  changes  that  addressed  user  training  were  made  to  the 
report  as  a  result  of  a  meeting  with  the  Director’s  staff. 

As  a  result  of  discussions  with  personnel  in  the  Office  of  the  Director,  Operational  Test 
and  Evaluation,  we  restated  the  recommendation  directed  to  the  Commanding  General, 
Army  Test  and  Evaluation  Command  to  ensure  that  ASAS  products  and  personnel  are 
ready  for  operational  testing.  Therefore,  we  request  that  the  Commanding  General,  Army 
Test  and  Evaluation  Command  and  the  Program  Executive  Officer,  Command,  Control, 
and  Communications  Tactical  comment  on  the  revised  recommendation  by 
December  9, 2003.  See  the  Finding  section  of  the  report  for  a  discussion  of  the 
management  comments  and  the  Management  Comments  section  of  the  report  for  the 
complete  text  of  the  comments. 
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Background 


The  All  Source  Analysis  System  (ASAS)  is  an  Army  Acquisition  Category  II 
mission-critical  weapon  system.  It  is  a  family  of  systems  and  is  the  Army’s 
primary  tactical  electronic  intelligence  system.  As  the  intelligence  and  electronic 
warfare  command  and  control  node  (cluster)  of  the  Army’s  Tactical  Command 
and  Control  System,  ASAS  interfaces  with  other  key  systems  in  the  Army  Battle 
Command  System.’  The  ASAS  filters  large  volumes  of  incoming  raw  data  and 
provides  intelligence  staff  elements  with  targeting  information,  predictions  on 
enemy  courses  of  action,  and  threat  and  force  protection  alerts  in  near  real  time  at 
collateral  and  compartmented  security  levels.  Variations  of  ASAS  have  been  in 
development  since  1979. 

Acquisition  Strategy.  Following  Office  of  Management  and  Budget  and 
DoD  guidance,  the  Army  is  acquiring  ASAS  component  systems  by  evolutionary^ 
blocks  with  spiral^  version  enhancements.  Each  block  and  version  builds  on 
previously  deployed  capabilities  and  functions.  Block  I,  deployed  between 
FY  1993  and  FY  1995,  provided  the  initial  capabilities  for  intelligence  analysis 
and  dissemination.  Block  II  development  began  in  FY  1994  and  will  enhance  the 
intelligence  fusion  capabilities  and  interoperability  of  Block  I  when  deployed  in 
its  final  version.  The  Milestone  C  acquisition  decision  for  ASAS  Block  II  is 
planned  for  April  2004.  Block  III  was  the  planned  objective  capability  for  ASAS. 
However,  the  Army  has  revised  the  ASAS  acquisition  strategy  and  plans  to 
replace  it  after  deployment  of  Block  II. 

Program  Management.  There  is  no  ASAS  Program  Management  Office; 
the  Program  Executive  Office,  Command,  Control  and  Communications  Tactical 
is  the  ASAS  Program  Executive  Office  and  its  milestone  decision  authority. 
Acquisition  management  and  life-cycle  support  is  a  team  effort.  The  Project 
Manager,  Intelligence  and  Effects  in  the  Program  Executive  Office  acquires,  tests, 
and  deploys  system  hardware  and  software  components,  in  coordination  with  the 
System  Manager,  Army  Training  and  Doctrine  Command  and  the  Army  Test  and 
Evaluation  Command.  The  Army  Communications  and  Electronics  Command 
assumes  life-cycle  support  of  ASAS  blocks  after  deployment.  Lockheed  Martin 
Mission  Systems,  Denver,  Colorado,  is  the  ASAS  Block  II  prime  contractor  and 
was  awarded  a  $247  million,  cost-plus-award-fee  contract  for  software 
development  and  integration  in  FY  1994. 

In  June  1999,  the  Army  redesignated  ASAS  from  an  Acquisition 
Category  I  to  an  Acquisition  Category  II  program.'’  As  a  result,  the  Army 


'The  Army’s  link  to  the  Global  Command  and  Control  System  that  receives  and  transmits  information  to 
the  joint  forces. 

^Evolutionary  acquisition  quickly  delivers  functional  capabilities  in  increments  and  allows  for  future 
improvements. 

^Spiral  development  is  an  evolutionary  acquisition  strategy  that  continually  refines  requirements  through 
demonstration  and  risk  management. 

"'Although  managers  of  Acquisition  Category  1  programs  are  required  to  report  program  status  to  DoD  and 
Congress  at  least  annually,  those  reports  are  generally  not  required  for  Acquisition  Category  II  programs. 
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provides  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  with  a  quarterly  progress  report;  however,  progress  is  not  comparatively 
measured  with  cost,  schedule,  and  performance  baselines  for  determining 
efficiency  indexes  and  projecting  results.  Total  Army  acquisition  development 
and  deployment  costs  for  AS  AS  from  inception  to  completion  could  exceed 
$2.5  billion.  From  FY  1983  through  FY  2002,  obligations  exceeded  $2  billion. 
Planned  obligations  for  ASAS  development  and  procurement  for  FY  2003  exceed 
$113  million. 

Remote  Workstation.  A  key  component  of  the  ASAS  family  of  systems 
is  the  Remote  Workstation,  which  is  a  combination  information  hardware  and 
software  system  that  provides  automated  support  to  intelligence  staff  elements 
and  other  designated  intelligence  organizations.  The  Remote  Workstation  is 
connected  to  the  existing  Intelligence  and  Electronic  Warfare  Battlefield 
Operating  System  and  the  Army  Battle  Command  System  architectures. 

Version  4  has  been  deployed,  and  Version  6  is  being  operationally  tested.  Since 
FY  1994,  the  Army  has  expended  more  than  $166  million  on  hardware  and 
software  products  for  the  Block  II  Remote  Workstation. 

The  General  Accounting  Office  has  listed  the  acquisition  of  DoD  weapon  systems 
as  a  high  risk  since  1990.  Two  prior  audits^  addressed  the  ASAS  Block  II 
Acquisition.  Both  reports  concluded  that  DoD  and  the  Army  prematurely 
commenced  deployment  of  Block  I  before  tests  demonstrated  that  the  system 
acquisition  met  operational  effectiveness  and  suitability  requirements.  The 
Inspector  General  of  the  Department  of  Defense  reported  that  milestone  reviews 
by  the  Defense  Acquisition  Board  were  inadequate  to  support  a  decision  to 
proceed  into  Block  II. 


Objective 


The  overall  audit  objective  was  to  evaluate  the  acquisition  management  of  the 
ASAS.  Specifically,  the  audit  determined  whether  the  ASAS  was  being  cost- 
effectively  acquired,  monitored,  tested,  and  prepared  for  deployment  and  system 
life-cycle  support  in  accordance  with  Office  of  Management  and  Budget  and  DoD 
guidance.  We  also  evaluated  the  management  control  program  related  to  the 
objective.  See  Appendix  A  for  a  discussion  of  the  audit  scope  and  methodology 
and  the  review  of  the  management  control  program. 


^General  Accounting  Office  Letter  Report,  GAO/NSIAD-94-49,  “  Army  Needs  to  Reconsider  and  Test  the 
All  Source  Analysis  System  Alternative,”  March  7,  1994,  and  Inspector  General  of  the  Department  of 
Defense  Report  No.  93-087,  “Review  of  the  All  Source  Analysis  System  as  a  Part  of  the  Audit  of  the 
Effectiveness  of  the  Defense  Acquisition  Board  Review  Process  -  FY  1993,”  April  20,  1993. 
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Acquisition  Management  of  the 
All  Source  Analysis  System 

The  Army  redesignated  the  ASAS  as  an  Acquisition  Category  II  program 
when  it  should  have  remained  an  Acquisition  Category  I  program. 

Further,  the  ASAS  Remote  Workstation  with  Version  4  software  did  not 
demonstrate  the  intended  results  when  it  was  operationally  tested.  Those 
conditions  occurred  because  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  and  the  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence),®  as  the 
functional  proponent  of  intelligence  systems  for  the  Office  of  Secretary  of 
Defense,  did  not  require  that  management  and  testing  controls  be 
implemented  when  the  system’s  acquisition  approach  was  redefined. 

Also,  the  Army  Test  and  Evaluation  Command  did  not  conduct  tests  in  a 
realistic  environment  with  skilled  military  intelligence  personnel,  and  the 
Director,  Operational  Test  and  Evaluation  did  not  report  test  results  and 
conclusions  in  accordance  with  section  2399,  title  10,  United  States  Code 
and  DoD  guidance.  As  a  result: 

•  The  Army  could  not  determine  whether  it  had  cost-effectively 
acquired,  monitored,  tested,  and  prepared  the  Remote 
Workstation  for  deployment  and  system  life-cycle  support, 

•  DoD  and  Army  independent  evaluators  unnecessarily 
expended  resources  on  a  system  that  required  additional 
hardware  and  software  development  and  user  training;  and 

•  Army  intelligence  units  had  to  apply  alternate  information 
solutions  to  compensate  for  missing  or  inoperable  system 
capability  when  the  deployed  ASAS  Block  II  Remote 
Workstation  did  not  achieve  the  intended  results. 


Guidance 


The  Office  of  Management  and  Budget  issued  guidance  addressing  management 
controls  in  Circular  A- 123,  “Management  Accountability  and  Control,’’  June  21, 
1995,  which  defines  management  controls  as  the  organization,  policies,  and 
procedures  used  to  reasonably  ensure  that: 

•  programs  achieve  their  intended  results; 

•  resources  are  used  consistent  with  mission; 


^Reorganized  as  the  Under  Secretary  of  Defense  (Intelligence)  and  the  Assistant  Secretary  of  Defense  for 
Networks  and  Information  Integration/DoD  Chief  Information  Officer. 
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•  programs  and  resources  are  protected  from  waste,  fraud,  and 
mismanagement; 

•  laws  and  regulations  are  followed;  and 

•  reliable  and  timely  information  is  obtained,  maintained,  reported,  and 
used  for  decision  making. 

Further,  Circular  A-109,  “Major  Systems  Acquisitions,”  April  1976,  states  that 
Federal  agencies  should  ensure  that  each  major  system,  such  as  ASAS,  fulfdls  a 
mission  need,  operates  effectively  in  its  intended  environment,  and  demonstrates 
a  level  of  performance  and  reliability  that  justifies  the  allocation  of  the  Nation’s 
limited  resources  for  its  acquisition  and  ownership.  DoD  Instruction  5010.40, 
“Management  Control  (MC)  Program  Procedures,”  August  28,  1996,  and 
DoD  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition  System,” 

May  12,  2003,  implement  Office  of  Management  and  Budget  guidance.  In 
addition.  Army  Regulation  73-1,  “Test  and  Evaluation  Policy,”  January  7,  2002, 
and  Army  Pamphlet  73-5,  “Test  and  Evaluation:  Operational  Test  and  Evaluation 
Guidelines,”  September  30,  1997,  reinforce  this  guidance  by  requiring  effective 
operational  tests  and  evaluations  for  system  acquisitions. 


ASAS  Management  Controls 


The  Under  Secretary  of  Defense  for  Acquisition  and  Technology’  did  not 
rigorously  apply  management  controls  for  the  acquisition  of  the  ASAS  Block  II 
and  its  key  component,  the  Remote  Workstation.  When  he  defined  the 
management  approach  to  the  Army  Tactical  Command  and  Control  System,^  the 
Under  Secretary  allowed  the  Army  to  redesignate  the  ASAS  as  an  Acquisition 
Category  II  program  and  to  delegate  oversight  to  its  materiel  developer  without 
concurrently  revising  requirements,  schedule,  and  performance  parameters  to 
coincide  with  the  revised  baseline  costs.  Further,  the  Under  Secretary  allowed  the 
Army  to  deploy  the  Remote  Workstation  with  Block  II  software  without 
conclusively  demonstrating  that  it  was  effective  and  suitable  in  an  operational 
environment. 

Acquisition  Thresholds  and  Oversight 

Section  2430,  title  10,  United  States  Code  defines  development  cost  thresholds 
for  major  systems  acquisitions.  In  turn,  DoD  applies  those  thresholds  to  prioritize 
its  acquisitions  and  assign  organizational  oversight.  As  gatekeepers,  overseers 
attempt  to  instill  discipline  and  accountability  into  major  acquisitions  by  requiring 
program  managers  to  periodically  report  on  cost,  schedule,  and  performance  and 
demonstrate  results  for  program  milestone  advancements. 


^Currently,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics. 

*The  Army  Tactical  Command  and  Control  System  includes  ASAS;  the  Advance  Field  Artillery  Tactical 
Data  System;  the  Forward  Area  Air  Defense  Command,  Control  and  Intelligence  System/Air  and  Missile 
Defense  Workstation;  the  Combat  Service  Support  Control  System;  and  the  Maneuver  Control  System. 
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Program  Oversight.  Prior  to  June  1999,  AS  AS  was  designated  an  Aequisition 
Category  I  program  with  oversight  responsibility  and  milestone  decision  authority 
assigned  to  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology.  As 
an  Acquisition  Category  I  program,  the  Army  was  required  to  measurably 
demonstrate  to  DoD  and  Congress  in  annual  Selected  Acquisition  Reports^  that  its 
capital  investment  in  the  ASAS  family  of  systems  was  progressing  according  to 
baseline  plans  and  meeting  milestone  goals.  However,  the  Selected  Acquisition 
Reports  ceased  when  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology,  in  a  December  10,  1998,  memorandum,  redirected  DoD  acquisition 
oversight  for  ASAS  and  four  other  systems  to  the  Army  Tactical  Command  and 
Control  System  (see  Appendix  B). 

Rebaseline  Direction.  In  addition  to  the  change  in  oversight,  the  Under 
Secretary’s  memorandum,  “Army  Tactical  Command  and  Control  System 
(ATCCS)  Management  Approach,”  directed  the  Army  to  work  with  the  Office  of 
the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  to  revise  the  baseline  for  the  ASAS  and  the  four  other  Army  Tactical 
Command  and  Control  System  Acquisition  Category  I  programs  into  separate 
program  blocks  to  synchronize  and  coordinate  them  with  software  releases  from 
the  Army  Battle  Command  System.  However,  ASAS  was  midway  through  its 
Block  II  system  acquisition  when  the  memorandum  requiring  synchronization  and 
coordination  was  released. 


Compliance  with  the  Rebaseline  Direction.  As  a  result  of  the  Under 
Secretary’s  memorandum,  the  Army  divided  costs  for  the  Block  II  ASAS  into  two 
parts.  Costs  accumulated  prior  to  January  1,  1999,  were  identified  as  Block  Ila 
and  costs  planned  after  December  31,  1998,  were  identified  as  Block  lib. 
However,  when  the  Army  revised  the  baseline  for  the  ASAS,  it  did  not  recognize 
$264  million  that  it  previously  invested  in  Block  Ila.  According  to  Army 
personnel,'^  the  revised  cost  baseline  for  ASAS  recognized  planned  costs  only 
after  December  31,  1998.  Further,  operational  requirements,  schedules,  key 
performance  parameters  and  budgets  were  not  revised  to  coincide  with  the 
reduced  baseline  cost.  As  a  result  of  those  differences,  the  Army  could  not 
determine  ASAS  progress,  evaluate  effectiveness  and  efficiency,  or  project 
program  results  because  management  control  baselines  for  the  revised  Block  II 
acquisition  were  incompatible  for  measuring  cost,  schedule,  and  performance. 
Also,  the  Army  subsequently  redesignated  the  ASAS  as  an  Acquisition  Category 
II  program  in  June  1999,  and  delegated  acquisition  oversight  to  the  Program 
Executive  Officer  for  Command,  Control  and  Communications  Systems' '  when 


^The  Selected  Acquisition  Report  is  a  comprehensive  status  report  that  is  required  by  section  2432,  title  10, 
United  States  Code  to  be  submitted  to  Congress  at  least  annually  for  all  Acquisition  Category  I  programs. 
It  provides  congressional  committees  with  key  cost,  schedule,  and  performance  information. 

'®In  the  absence  of  documentation,  we  relied  on  discussions  with  personnel  involved  with  ASAS 
development  at  the  Offices  of  the  Program  Executive  Officer,  Command,  Control  and  Communications 
Tactical  and  the  Project  Manager,  Intelligence  and  Effects. 

"Subsequently  renamed  Program  Executive  Officer  for  Command,  Control  and  Communications  Tactical. 
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Block  II  development  costs  slipped  below  the  Acquisition  Category  I  threshold 
for  research,  development,  test  and  evaluation.'^ 

Remote  Workstation  Users 

Army  intelligence  units  began  receiving  Remote  Workstations  with  Block  II, 
Version  4  software  in  FY  2000.  To  determine  whether  the  Remote  Workstation 
supported  mission  requirements,  we  visited  deployed  sites  and,  from  user 
interviews,  documented  observations  on  the  system’s  operational  effectiveness 
and  suitability'^  at  five  Army  installations  and  three  National  Guard  locations. 
Users  generally  agreed  that  the  Remote  Workstation  was  a  useful  planning  tool; 
however,  they  also  believed  that  the  Remote  Workstation  had  operational 
effectiveness  and  suitability  limitations.  Specifically,  at  all  sites  visited  users 
stated  that: 

•  the  system  was  unable  to  maintain  around-the-clock  operations, 

•  training  was  inadequate  and/or  infrequent, 

•  certain  environmental  conditions  affected  hardware  operations, 

•  system  operations  were  overly  dependent  on  contractor  support,  and 

•  the  system  lacked  multi- Service  interoperability. 

Further,  intelligence  units  with  deployed  Block  II  Remote  Workstations  with 
Version  4  software  had  to  compensate  for  missing  and  inoperable  information 
system  capability  with  alternative  solutions,  such  as  Power  Point  presentations, 
map  boards,  and  radios  and  telephones  because  the  Remote  Workstation  could  not 
support  intelligence  collecting,  analysis,  and  targeting  functions.  Appendix  C 
lists,  by  topic,  general  and  specific  observations  made  by  users. 

Test  and  Evaluation 

The  Army  conditionally  deployed  the  Remote  Workstation  with  Block  II, 

Version  4  software  when  test  results  and  evaluations  did  not  conclusively 
demonstrate  that  the  Remote  Workstation  was  effective  and  suitable  in  an 
operational  environment.  Contrary  to  Office  of  Management  and  Budget,  DoD, 
and  Army  guidance,  the  Army  did  not  demonstrate  that  the  Remote  Workstation, 
was  effective  and  suitable  when  operated  in  a  realistic  environment  by  skilled 
military  intelligence  personnel. 


'^Acquisitions  totaling  more  than  $355  million  in  research,  development,  test  and  evaluation  expenditures 
were  identified  as  Acquisition  Category  I  programs. 

'^Army  Regulation  73-1,  “Test  and  Evaluation  Policy,”  January  7,  2002,  defines  operational  suitability  as 
“The  degree  to  which  a  system  can  be  satisfactorily  placed  in  field  use  with  consideration  given  to 
availability,  compatibility,  transportability,  interoperability,  reliability,  wartime  usage  rates, 
maintainability,  safety,  human  factors,  manpower  supportaljility,  logistics  supportability,  and  training 
requirements.” 
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The  Under  Seeretary  of  Defense  for  Acquisition  and  Technology,  in  his 
December  10,  1998,  memorandum,  granted  the  Army  a  waiver  that  effectively 
reduced  the  deployment  threshold  for  ASAS  operational  tests.  The  Under 
Secretary  allowed  the  Army  to  deploy  ASAS  components  if  test  and  training 
events  confirmed  that  functionality  was  migrating  toward  Block  111  requirements, 
rather  than  satisfying  operational  requirements.  However,  our  review  of  the 
Army  operational  test  report,'"^  as  well  as  a  report  prepared  for  the  Director, 
Operational  Test  and  Evaluation  by  a  Federally  Funded  Research  and 
Development  Center,'^  concluded  that  tests,  results,  and  evaluations  for  the 
Remote  Workstation  did  not  demonstrate  system  functionality  in  an  operational 
environment. 

Quality  of  Tests.  The  Army  did  not  conduct  operational  tests  in  a  realistic 
environment  in  accordance  with  Army  Pamphlet  73-5.  Instead  of  conducting  a 
single  operational  test  for  measuring  effectiveness  and  suitability,  the  Army 
conducted  two  separate  and  distinct  tests.  One  limited  user  test,  conducted  in  a 
controlled  environment,  demonstrated  technical  capability,  and  the  other  limited 
user  test,  conducted  during  a  scheduled  warfighting  exercise,  demonstrated 
operational  performance.  Neither  test  demonstrated  operational  effectiveness  and 
suitability  in  a  realistic  environment. 

The  Army,  in  the  technical  capability  test,  interfaced  Remote  Workstations  with 
the  Army  Tactical  Command  and  Control  System  on  the  local  area  network  rather 
than  using  redundant  communication  and  data  feeds  in  accordance  with  the 
system’s  Operational  Requirements  Document.  Further,  The  Army  also  did  not 
test  production  representative  system  configurations.  The  Army  updated  software 
between  the  two  exercises  and  tested  hardware  that  did  not  include  production 
representative  processors.  Also,  more  contractor  support  staff  participated  in  tests 
than  will  support  the  Remote  Workstation  in  the  field.  As  a  result,  the  Army  did 
not  conduct  tests  with  tactics,  doctrine,  logistics,  and  maintenance  support  in 
accordance  with  Army  guidance. 

Test  Results.  Test  results  for  the  Remote  Workstation  did  not  decisively 
demonstrate  that  the  system  was  migrating  towards  objective  requirements.  Users, 
demonstrating  effectiveness  and  suitability  experienced  difficulties  analyzing 
information  and  operating  system  hardware  and  software.  Contrary  to  the  Army’s 
evaluation,  test  results  demonstrated  that  Remote  Workstation  users  were  unable 
to  perform  substantive  intelligence  functions  when  compared  with  the  Critical 
Operational  Issues  and  Criteria  (COICs)  identified  in  the  ASAS  Test  and 
Evaluation  Master  Plan. 

COICs.  COICs  provide  objective  and  subjective  benchmarks  for 
measuring  test  results  and  evaluating  mission  performance.  They  address  the 
functional  requirements  for  determining  effectiveness  and  training  readiness, 
deployability,  sustainability,  and  survivability.  According  to  Army  Regulation 


'"'“System  Evaluation  Report  for  the  All  Source  Analysis  System  (ASAS)  Block  II  Remote  Workstation 
(RWS),”  Army  Operational  Test  and  Evaluation  Command,  September  1999. 

'^“Operational  Testing  of  the  ASAS  Remote  Workstation,”  Institute  for  Defense  Analyses  Paper  P-3490, 
November  12,  1999,  performed  for  the  DoD  Director,  Operational  Test  and  Evaluation. 
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73-1,  “Test  and  Evaluation  Policy,”  COICs  must  be  answered  before  systems  are 
ready  to  enter  full-rate  production  for  deployment. 

The  Army’s  decision  to  conditionally  deploy  the  Remote  Workstation  was 
based  on  four  COICs.  COICs  1  and  2  measured  functional  effectiveness,  and 
COICs  3  and  4  measured  system  suitability.  The  key  effectiveness  and  suitability 
issues  were: 

•  Does  the  system  satisfy  the  Commander’s  intelligence  and  targeting 
support  requirements?  (COIC  1) 

•  Does  the  system  establish  and  maintain  interfaces  with  other  systems 
to  provide  required  information  exchanges?  (COIC  2) 

•  Does  the  system  deploy  and  operate  on  the  battlefield?  (COIC  3) 

•  Does  the  system  provide  a  wartime  capability  for  sustained  combat 
operations  with  the  level  of  system  training  and  readiness  during 
peacetime  for  operators/analysts/maintainers?  (COIC  4) 

Criteria.  The  Army  selected  76  effectiveness  and  suitability  criteria  for 
testing  and  evaluating  whether  the  Remote  Workstation  was  migrating  towards 
Block  III  requirements.  Forty-four  criteria  measured  functional  effectiveness  and 
32  criteria  measured  suitability.  However,  the  Army  did  not  test  and  evaluate  all 
the  selected  criteria.  Thirty-one  criteria  were  deferred  or  not  evaluated  because 
the  Army  believed  that  tests  required  Remote  Workstation  interoperability  and 
interfaces  with  other  ASAS  components  and  support  systems.  The  following 
tables  show  the  results  of  the  Army’s  effectiveness  and  suitability  criteria 
evaluations  for  the  Remote  Workstation. 


Table  1.  Effectiveness  Evaluations 

(in  percent) 


Criteria  Criteria 

Criteria  Met  Partially  Met  Not  Met 

36  25  16 

(16  of  44)  (11  of  44)  (7  of  44) 


Criteria 

Deferred 

23 

(10  of  44) 


Criteria  Not 
Evaluated 
0 

(0  of  44) 


Table  2.  Suitability  Evaluations 

(in  percent) 


Criteria  Met 
28 

(9  of  32) 


Criteria 
Partially  Met 
3 

(1  of  32) 


Criteria 
Not  Met 
3 

(1  of  32) 


Criteria 

Deferred 

50 

(16  of  32) 


Criteria  Not 
Evaluated 
16 

(5  of  32) 
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Evaluations  of  Effectiveness.  The  Army’s  evaluations  of  system 
effectiveness  showed  that  61  percent  (27  of  44 y  of  the  test  results  met  or 
partially  met  criteria.  However,  we  believe  that  even  those  results  were 
overstated  and  not  balanced  because  comparisons  between  criteria  and  test  results 
for  the  criteria  met  and  partially  met  categories  did  not  recognize  the  offsetting 
effects  of  the  user’s  inability  to  operate  the  system  and  reliance  on  support 
contractor  assistance. 


Using  the  Army’s  criteria  for  measuring  effectiveness,  we  derived 
different  results  for  1 1  criteria  that  the  Army  categorized  as  “Criteria  Met’’  and 
“Criteria  Partially  Met.”  We  believe  that  only  36  percent'^  (16  of  44)  of  the  tests 
met  or  partially  met  criteria.  Criteria  test  differences  between  the  Army  and  the 
audit  evaluations  of  Remote  Workstation  effectiveness  can  be  found  in 
Appendix  D.  The  following  three  examples  explain  why  the  audit  team’s 
evaluations  differ  with  the  Army’s  “Criteria  Partially  Met”  evaluations. 


Criterion  I.2.I.4.  Criterion  1.2. 1.4  requires  analysts  and 
operators  to  provide  complete  answers  to  more  than  85  percent  of  priority 
intelligence  requirements  and  answers  to  all  requests  for  information.  However, 
when  the  Army  tested  this  criterion,  analysts  and  operators  provided  complete 
answers  to  only  12  percent  (2  of  17)  of  the  priority  intelligence  requirements  and 
did  not  answer  all  requests  for  information.  Rather  than  attribute  the  results  to  the 
tested  Remote  Workstation,  the  Army  attributed  the  below-threshold  performance 
for  providing  answers  to  requirements  and  requests  to  users  who  were  unable  to 
operate  the  Remote  Workstation;  the  absence  of  an  operations  staff  to  guide  the 
intelligence  process;  and  ineffective  or  nonexistent  tactics,  techniques  and 
procedures. 


Criterion  I.2.3.5.  Criterion  1.2. 3. 5  requires  analysts  and 
operators  to  nominate  70  percent  of  the  unique  high  value  and  high  payoff  targets 
that  meet  target  criteria.  However,  when  the  Army  tested  this  criterion,  the 
analysts  and  operators  identified  only  28  percent  (55  of  195)  of  high  payoff 
targets.  The  below-threshold  performance  was  not  due  to  the  Remote 
Workstation,  according  to  the  Army,  but  to  the  inability  of  analysts  and  operators 
to  operate  the  system.  The  Army  indicated  that  a  separate  test  achieved  the 
desired  goal  when  support  contractors  operated  the  Remote  Workstation  in  a 
nonoperational  environment. 


Criterion  I.2.4.8.  Criterion  1. 2.4.8  requires  analysts  and 
operators  to  prepare,  edit,  and  transmit  and  disseminate  a  selected  set  of  collection 
management  messages  and  graphic  products.  However,  because  of  system 
instability,  difficulty  of  use,  and  lack  of  shared  capability,  users  were  unable  to 


'^Seventy-nine  percent  (27  of  34)  of  the  test  results  met  or  partially  met  effectiveness  criteria  when 
deferred  criteria  were  eliminated  from  the  determinations. 

'^Forty-seven  percent  (16  of  34)  of  the  test  results  met  or  partially  met  effectiveness  criteria  when  deferred 
criteria  were  eliminated  from  the  determinations. 
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perform  the  complete  function.  Analysts  and  operators  had  to  find  alternate 
solutions  for  the  message  preparation  and  editing  processes.  Despite  these 
functional  limitations,  the  Army  believed  that  the  Remote  Workstation  partially 
met  this  criterion  because  it  could  transmit  and  disseminate  selected  sets  of 
collection  management  messages  and  graphic  products  when  data  were  placed 
into  the  system  from  the  alternate  sources. 


Evaluations  of  Suitability.  The  Army’s  evaluations  for  suitability 
showed  that  91  percent  (10  of  11)'^  of  test  results  met  or  partially  met  the  criteria. 
However,  we  believe  that  the  Army’s  results  were  overstated  because  users 
demonstrating  the  system  had  to  rely  on  contractor  assistance  due  to  technical 
issues  and  operators  who  were  not  sufficiently  trained  to  operate  the  system.  We 
believe  that  60  percent  (6  of  10)  of  Army  criteria  met  or  criteria  partially  met 
evaluations  were  overstated.  Criteria  test  differences  between  the  Army  and  audit 
evaluations  for  logistics  support,  software  maturity,  system  stability  and 
robustness,  subsystem  interoperability,  and  the  ability  of  representative  soldiers  to 
perform  critical  functions  and  maintain  and  support  the  Remote  Workstations  in 
an  operational  environment  can  be  found  in  Appendix  E. 


Effectiveness  and  Suitability.  The  Remote  Workstation  did  not  demonstrate  that 
it  could  provide  commanders  with  qualitative  intelligence  data  to  support  military 
operations.  Army  Field  Manual  34-1,  “Intelligence  and  Electronic  Warfare 
Operations,’’  September  27,  1994,  requires  that  military  intelligence  be  timely, 
relevant,  accurate,  and  synchronized  to  support  tactical,  operational,  and  strategic 
commanders  across  the  range  of  military  operations.  Because  the  Army 
overstated  the  test  results  for  hardware  and  software  effectiveness  and  suitability 
from  nonrepresentational  tests,  and  deferred  or  did  not  test  more  than  40  percent 
(31  of  76)  of  the  required  effectiveness  and  suitability  criteria,  the  Army  did  not 
convincingly  demonstrate  that  the  Remote  Workstation  was  migrating  towards 
Block  III  requirements. 

Operational  Testing  Requirements 

The  Army  and  the  Director,  Operational  Test  and  Evaluation  did  not  adequately 
comply  with  requirements  for  operational  testing.  The  Remote  Workstation  with 
Version  4  software  was  allowed  to  be  deployed  when  operational  test  results 
demonstrated  that  the  system  required  additional  development  and  that  users 
required  additional  training  prior  to  testing. 

Army  Operational  Testing  and  Evaluation.  The  Army  had  not  prepared  the 
Remote  Workstation  or  personnel  demonstrating  the  system  for  operational 
testing  and  evaluation.  Army  Regulation  73-1  requires  that  only  operationally 
effective,  suitable,  and  survivable  systems  are  delivered  to  users.  Further, 
operational  test  readiness  is  confirmed  after  systems  have  been  stressed  to  at  least 
the  levels  expected  in  the  operating  environment,  and  demonstrate  a  level  of 
achievement  of  system  performance,  safety,  health  hazards,  survivability,  human 


After  eliminating  deferred  criteria  from  determinations  of  system  suitability. 
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factors  engineering,  reliability,  availability,  maintainability,  and  integrated 
logistics  support.  Based  on  the  Army  Test  and  Evaluation  Command’s  test  results 
and  evaluations  for  the  Remote  Workstation,  we  believe  that  the  Army  had  not 
prepared  the  Remote  Workstation  for  operational  testing  and  unnecessarily 
expended  resourees  on  a  system  that  required  additional  development  and  user 
training  prior  to  testing.  Army  Pamphlet  73-5  states  that,  in  the  past,  where  tests 
were  eliminated  or  redueed,  deficient  systems  were  deployed,  resulting  in  costly 
modifications.  However,  when  testing  is  adequate  and  complete,  a  system  is 
deployed  with  favorable  results  in  the  field. 

Director,  Operational  Test  and  Evaluation.  The  Direetor,  Operational  Test  and 
Evaluation  expressed  a  qualified  opinion  in  its  FY  1999  Annual  Report  that  the 
Remote  Workstation  did  not  fully  meet  the  requirements  for  effeetiveness  and 
suitability,  as  defined  by  the  Army’s  Critieal  Operational  Issues.  Seetion  2399, 
title  10,  United  States  Code  and  DoD  Instruction  5000.2  require  that  the  Director, 
Operational  Test  and  Evaluation  express  an  opinion  on  whether  tests  and 
evaluations  were  adequate  and  whether  the  results  eonfirmed  that  tested  items  or 
components  were  effeetive  and  suitable  for  combat.  However,  instead  of  stating 
an  unqualified  opinion  addressing  the  quality  of  tests  and  results,  the  Director 
concluded  that  the  Remote  Workstation  should  be  fielded,  despite  software, 
interoperability,  and  demonstrated  logistics  supportability  problems,  because  it 
provided  operational  value.  We  believe  that  the  eonelusion  was  inconsistent  with 
the  Direetor’ s  responsibility  for  assessing  operational  tests  and  subsequent  results, 
because  it  did  not  address  the  system’s  effectiveness  and  suitability  for  combat. 


Conclusion 


The  Under  Seeretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  in 
coordination  with  the  Under  Seeretary  of  Defense  (Intelligence),  needs  to 
reeonsider  the  decision  to  redirect  acquisition  oversight  for  ASAS  to  the  Army 
Taetieal  Command  and  Control  System.  That  decision  allowed  the  Army  to 
revise  the  baseline  for  program  costs  without  concurrently  adjusting  sehedule  and 
performance  baselines;  to  redesignate  the  program’s  aequisition  eategory;  and, 
subsequently,  to  delegate  oversight  responsibilities  and  milestone  deeision 
authority  to  the  ASAS  Program  Executive  Offieer.  The  deeision  redueed  testing 
thresholds  and  allowed  the  Army  to  deploy  ASAS  components  that  did  not 
always  satisfy  user  requirements. 

In  addition,  the  Army  Test  and  Evaluation  Command’s  test  and  evaluation  results 
did  not  validate  that  the  Remote  Workstation  developed  by  the  Program 
Exeeutive  Officer,  Command,  Control  and  Communieations  Taetieal  was  ready 
for  operational  testing;  and  the  Direetor,  Operational  Test  and  Evaluation 
expressed  a  qualified  opinion  on  whether  tests  and  evaluations  were  adequate  and 
whether  the  test  results  confirmed  system  effeetiveness  and  suitability. 

As  a  result  and  as  identified  in  prior  audits,  the  Army  could  not  determine 
whether  ASAS  and  a  key  component,  the  Remote  Workstation  with  Version  4 
Software,  were  eost-effeetively  aequired,  monitored,  tested,  and  prepared  for 
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deployment  and  system  life-cycle  support'^  in  accordance  with  Office  of 
Management  and  Budget,  DoD,  and  Army  guidance.  In  addition,  Army 
intelligence  units  with  Remote  Workstations  had  to  apply  alternate  information 
solutions  to  compensate  for  missing  and  inoperable  system  capability,  and  DoD 
and  Army  independent  evaluation  organizations  unnecessarily  expended 
resources  on  an  ASAS  component  that  required  additional  development  and  user 
training  prior  to  testing. 


Recommendations,  Management  Comments  and  Audit 
Response 


Revised  Recommendation.  We  revised  Recommendation  5  in  response  to 
suggestions  by  the  Director,  Operational  Test  and  Evaluation  at  a  meeting  with 
his  staff.  Specifically,  the  recommendation  directs  the  Commanding  General, 
Army  Test  and  Evaluation  Command  and  the  Program  Executive  Officer, 
Command,  Control  and  Communications  Tactical  to  jointly  ensure  that  All 
Source  Analysis  System  products  are  adequately  tested  and  users  demonstrating 
system  products  are  adequately  trained. 

1.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics: 

a.  Redesignate  the  All  Source  Analysis  System  as  an  Acquisition 
Category  I  Major  Defense  Acquisition  Program  in  accordance  with  DoD 
guidance. 

b.  Assume  responsibility  for  providing  DoD  acquisition  management 
oversight  for  the  All  Source  Analysis  System  Block  II  acquisition. 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
Comments.  The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  nonconcurred,  although  he  agreed  in  principle  with  the 
recommendation.  Rather  than  immediately  upgrading  the  All  Source  Analysis 
System  to  an  Acquisition  Category  I,  the  Under  Secretary  believed  that  it  would 
be  appropriate  to  reevaluate  the  system  acquisition  when  the  Major  Defense 
Acquisition  Program  List  is  updated  in  December  2003.  In  addition,  if  the 
weapon  system  meets  the  dollar  thresholds  for  a  Major  Defense  Acquisition 
Program  and  is  elevated  to  a  Acquisition  Category  I  program,  management 
oversight  would  be  provided. 

Audit  Response.  Management  comments  meet  the  intent  of  the  recommendation 
to  reevaluate  the  ASAS  acquisition.  We  believe  that  the  ASAS  Block  II 
acquisition  meets  the  Major  Defense  Acquisition  Program  threshold  and  is  an 
Acquisition  Category  I  program. 


'^Acquisition  system  life-cycle  support  for  Army  acquisitions  concludes  after  deployment. 
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2.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  in  coordination  with  the  Under  Secretary  of 
Defense  (Intelligence),  raise  the  deployment  threshold  for  the  All  Source 
Analysis  System  operational  tests  from  “migrating  towards  objective 
requirements”  to  satisfying  benchmarks  for  critical  operational  issues  and 
criteria. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Comments.  The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  concurred,  and  stated  that  guidance  will  be  issued  if  the  All  Source 
Analysis  System  is  elevated  to  an  Acquisition  Category  I  program. 

3.  We  recommend  that  the  Program  Executive  Officer,  Command,  Control 
and  Communications  Tactical  restore  baseline  integrity  to  the  All  Source 
Analysis  System  by  ensuring  that  cost,  schedule,  and  performance 
parameters  measure  progress,  determine  effectiveness  and  efficiency  indexes, 
and  project  program  results. 

Program  Executive  Officer,  Command,  Control  and  Communications 
Tactical  Comments.  The  Program  Executive  Officer,  Command,  Control  and 
Communications  Tactical  concurred  and  will  implement  the  recommendation  by 
October  15,  2003. 

4.  We  recommend  that  the  Director,  Operational  Test  and  Evaluation,  when 
evaluating  the  quality  of  the  All  Source  Analysis  System  operational  tests 
and  analyzing  subsequent  results,  express  an  unqualified  opinion  on  whether 
the: 


a.  Tests  and  evaluations  are  adequate,  and 

b.  Results  confirm  that  items  or  components  are  effective  and 
suitable  for  combat. 

Director,  Operational  Test  and  Evaluation  Comments.  The  Director  of 
Operational  Test  and  Evaluation  concurred  and  stated  that  he  would  provide  an 
operational  effectiveness,  survivability,  suitability,  and  test  adequacy  assessment 
of  the  ASAS  Block  II  when  the  Army  completes  its  ASAS  initial  operational  test 
and  evaluation  in  October  2004. 

Audit  Response.  We  request  that  the  Director,  Operational  Test  and  Evaluation 
provide  us  a  copy  of  his  assessment  when  completed. 

5.  We  recommend  that  the  Commanding  General,  U.S.  Army  Test  and 
Evaluation  Command,  in  coordination  with  the  Program  Executive  Officer, 
Command,  Control  and  Communications  Tactical,  comply  with  Army 
Regulation  73-1,  “Test  and  Evaluation  Policy,”  January  7,  2002,  by  ensuring 
that: 


a.  All  Source  Analysis  System  products  are  stressed  to  levels 
expected  in  an  operating  environment,  and 
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b.  Personnel  demonstrating  the  All  Source  Analysis  System  are 
sufficiently  trained  to  operate  system  products  and  analyze  resulting 
information  in  accordance  with  operational  requirements. 

Management  Comments  Required.  Management  Comments  from  the 
Commanding  General,  Army  Test  and  Evaluation  Command  were  received  too 
late  to  be  incorporated  into  the  final  report.  However,  we  revised  this 
recommendation  to  ensure  that  ASAS  products  and  personnel  are  ready  for 
operational  testing.  Accordingly,  we  request  that  the  Commanding  General, 
Army  Test  and  Evaluation  Command  and  the  Program  Executive  Officer, 
Command,  Control  and  Communications  Tactical  comment  on  the  revised 
recommendation  by  December  9,  2003. 
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Appendix  A.  Scope  and  Methodology 


We  reviewed  doeumentation  dated  from  Mareh  1995  through  May  2003  and 
interviewed  officials  in  the  Office  of  the  Secretary  of  Defense  and  Army  and 
contractor  personnel.  To  accomplish  the  audit  objectives,  we: 

•  Interviewed  officials  and  obtained  documentation  from  the  Offices  of 
the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence);  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics;  the  Assistant  Secretary  of 
the  Army  (Acquisition,  Logistics,  and  Technology);  the  Program 
Executive  Officer  Command,  Control  and  Communications  Tactical; 
the  Project  Manager,  Intelligence  and  Effects;  the  Army  Training  and 
Doctrine  Command  System  Manager  for  AS  AS;  the  Army  Test  and 
Evaluation  Command;  the  Army  Intelligence  Center  and  Fort 
Huachuca;  the  Institute  for  Defense  Analysis;  the  Arctic  Slope 
Regional  Corporation;  and  Sytex,  Incorporated,  to  identify  system 
background,  history,  and  reasons  for  acquisition  decisions. 

•  Reviewed  program  documents  including  the  Operational 
Requirements  Document,  Change  3,  March  1997;  the  Acquisition 
Strategy  Report,  September  2000;  and  the  ASAS  Test  and  Evaluation 
Master  Plan,  May  2000,  to  determine  whether  the  project  office 
developed  and  implemented  an  acquisition  strategy  and  a  test  and 
evaluation  master  plan  that  were  linked  to  user  requirements. 

•  Reviewed  contract  documents  for  Lockheed  Martin  Mission  Systems 
and  Austin  Information  Systems;  cost  performance  reports,  February 
2002;  and  monthly  acquisition  program  review  reports  to  identify 
work  performed  by  contractors  and  to  determine  the  adequacy  of  the 
Army’s  oversight  of  ASAS  contractors. 

•  Reviewed  ASAS  life-cycle  cost  estimates  conducted  in  1997  and  2002 
to  determine  total  life-cycle  costs,  total  acquisition  costs,  and  the  basis 
of  support  for  the  cost  estimates. 

•  Reviewed  ASAS  funding  summaries  for  FYs  1983  through  2002,  and 
FY  2003  and  FY  2004  budget  documents  to  determine  the  investment 
in  the  program. 

•  Analyzed  ASAS  Research,  Development,  Test  and  Evaluation  Budget 
Item  Justification  Sheets  (R-2  Exhibits)  for  FYs  1996  through  2002  to 
determine  whether  the  split  in  the  ASAS  Block  II  Development  Costs 
was  reflected  in  budget  submissions. 

•  Analyzed  the  ASAS  Acquisition  Program  Baseline  from  1995  to  2002 
to  identify  baseline  trends. 
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•  Reviewed  the  ASAS  Selected  Acquisition  Report  for  the  year-ended 
December  1998  to  identify  development  costs  for  ASAS  Block  IIA 
and  Block  IIB. 

•  Analyzed  the  “System  Evaluation  Report  (SER)  for  the  All  Source 
Analysis  System  (ASAS)  Block  II  Remote  Workstation  (RWS),” 

Army  Operational  Test  and  Evaluation  Command,  September  1999,  to 
determine  whether  operational  tests  and  evaluations  performed  were 
adequate,  and  whether  test  and  evaluation  results  confirmed  that 
deployed  items  and  components  were  effective  and  suitable  for  combat 
in  accordance  with  section  2399,  title  10,  United  States  Code  and  DoD 
guidance. 

•  Reviewed  a  DoD  contractor-prepared  report,  “Operational  Testing  of 
the  ASAS  Remote  Workstation,”  by  the  Institute  for  Defense  Analyses 
to  evaluate  the  DoD  assessment  of  the  ASAS  test  program  as 
prescribed  by  section  2399,  title  10,  United  States  Code  and  DoD 
guidance. 

•  Used  24  standard  questions  derived  from  the  Test  and  Evaluation 
Master  Plan  and  interviewed  165  users  and  6  field  support  contractors 
to  document  observations  on  the  operational  effectiveness  and 
suitability  of  the  Remote  Workstation  with  Version  4  software. 
Interviews  took  place  at  Fort  Gordon,  Georgia;  Fort  Campbell, 
Kentucky;  Fort  Bliss,  Texas;  Fort  Hood,  Texas;  Fort  Lewis, 
Washington,  Camp  Mabry,  Texas;  Camp  Murray,  Washington;  and 
Kent  Armory,  Washington. 

We  performed  this  audit  from  July  2002  through  June  2003  in  accordance  with 
generally  accepted  government  auditing  standards. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Use  of  Technical  Assistance.  We  did  not  use  technical  assistance  to  perform  this 
audit. 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  DoD.  This  report  provides  coverage  of 
the  Weapons  Acquisition  Process  and  the  Information  Technology  Investment 
high-risk  areas. 


Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26,  1996, 
and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program  Procedures,” 
August  28,  1996,  require  DoD  organizations  to  implement  a  comprehensive 
system  of  management  controls  that  provides  reasonable  assurance  that  programs 
are  operating  as  intended  and  to  evaluate  the  adequacy  of  those  controls. 
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Scope  of  the  Review  of  the  Management  Control  Program.  In  accordance 
with  DoD  policy,  acquisition  managers  are  to  use  program  cost,  schedule,  and 
performance  parameters  as  control  objectives  to  implement  the  requirements  of 
DoD  Directives  5010.38.  Accordingly,  we  limited  our  review  to  management 
controls  in  the  Office  of  the  Secretary  of  Defense  and  the  Army  that  were  directly 
related  to  program  cost,  schedule,  and  performance. 

Adequacy  of  the  Management  Controls.  Management  controls  were 
inadequate.  We  identified  material  management  control  weaknesses  in  the  Office 
of  the  Secretary  of  Defense  and  the  Army,  as  defined  by  DoD 
Instruction  5010.40,  “Management  Control  (MC)  Program  Procedures.”  August 
28,  1996.  The  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
decreased  management  controls  when  he  allowed  the  Army  to  recategorize  the 
ASAS  from  an  Acquisition  Category  I  to  an  Acquisition  Category  II  program,  and 
thereby  eliminated  Selected  Acquisition  Reports  for  measuring  cost,  schedule, 
and  performance  progress.  In  addition.  Army  baselines  for  measuring  cost, 
schedule,  and  performance  could  not  be  compared  to  determine  ASAS  progress, 
evaluate  effectiveness  and  efficiency,  and  project  program  results.  Further,  by 
reducing  operational  testing  thresholds,  the  Under  Secretary  allowed  the  Army, 
with  support  from  independent  testers,  to  deploy  ASAS  products  without 
demonstrating  operational  effectiveness  and  suitability.  In  addition,  we  believe 
that  DoD  and  Army  independent  evaluators  did  not  adequately  comply  with  laws 
and  regulations  when  reporting  and  reviewing  operational  test  results  from  a 
system  that  required  additional  development.  If  implemented,  the 
recommendations  will  correct  the  identified  weaknesses.  We  will  provide  a  copy 
of  this  report  to  the  senior  officials  responsible  for  management  controls  in  the 
Office  of  the  Secretary  of  Defense  and  the  Army. 

Adequacy  of  Management’s  Self-Evaluation.  Officials  in  the  Project 
Management  Office,  Intelligence  and  Effects  did  not  identify  the  ASAS  as  an 
assessable  unit  and  therefore  did  not  identify  or  report  the  material  management 
control  weaknesses  identified  by  the  audit. 


Prior  Coverage 


No  prior  coverage  has  been  conducted  on  the  acquisition  management  of  the 
Army  All  Source  Analysis  System  during  the  last  5  years. 
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Appendix  B.  Under  Secretary  Of  Defense 

Memorandum,  December  10, 1998 


memorandum  for  distribution 

SUBJECT:  ARMY  TACTICAL  COMMAND  AND  CONTROL  SYSTEM  (ATCCS) 

management  approach 

As  foUow-iip  v>  [he  Apnl  159S  DAES  Agrek^nieDTS  and  Assignmnus  memo  (otcaelutUDtL  I 
teviewed  ihn  proposed  managemenz  approach  R>r  the  Army  Taciical  Command  and  Control 
System  (ATCCS)  on  October  29,  1 99S-  Based  on  ihe  R^Uowing  guidance,  I  a^rovc  die 
management  approach  to  fbtms  OSD  oversigbr  at  the  system-of-s>stem«  level,  la-baseline  ihe 
ihte  ATCC5  ACaT  i  programs,  comlnuje  ^iral  developmeoi,  and  odopT  a  modular  contracting 
KppTOJLch  fbr  dusc  programs. 

For  the  near  mim,  the  Army  and  OSD  will  implement  the  system-of-systems  oversight 
approach  by  leportmg  and  assessing  ar  the  level  of  ATCCS  and  its  iniei^ra  to  other  key 
sysxms  in  the  Army  Baide  Command  System  (ABCS).  The  Army  wdl  repoei  on  the  system 
of-syavems  id  a  Quanerly  Integrated  Summary,  a  tailored  DAE5-li)^  narouive  lepoit,  beginning 
in  January  1999.  The  report  will  include  (IJ  insi|^  into  system^of-systems  relialnli^^  fiKiiTitig 
on  how  reliabiUcy  improves  over  time^  and  how  it  changes  as  system-of-^systems  functionality 
changes;  and  (2)  progress  in  achieving  jodor  inieroperahiltty  roquiremenrs  of  the  digitized 
batdeEeld,  cftiphasizing  die  starus  of  internes  with  odicr  Services  and  Allies^  OSD,  wldt  the 
Aimy,  will  establish  benchmarks  for  assessing  the  smnenuy,  using  exisciiif  Army  documents 
such  as  the  ABCS  Capstone  Requiremenu  Docutneni,  the  Imegrated  System  E^uumn  Master 
Flan,  and  the  individual  Operaticmal  Requirements  DocumeDts  of  the  Batdedeld  Funcdonal  Areas 
where  praciicaL  OSD  will  provide  quarterly  feedback  on  the  appropriaie  eonrent  of  Ttri^:  report 
dirou^  the  C3t  DaES  points  of  conracx  to  the  Anny.  jibiie  OSD  oversighi  will  he  at  the 
systen^of-sy stems  levcL  the  Army  wf  1!  maintain  an  acquisition  managemenr  process  for  each 
individual  system. 

For  the  long  term,  OSD,  in  coordinatton  with  the  Army,  will  deteronne  the  most  appropnaie 
system^of-systems  level  ^  oversight-  The  objective  for  OSD  oversight  is  a  sysxem-of-sysmms 
level  (1)  tbai  encampasse^  die  ATCCS  systEuns  and  linkages  to  die  other  C2  and  caromunicatioDS 
systems  ihac  together  achieve  the  digitized  battlefield,  and  (?)  with  natural  bencbinaiks  such  as  a 
requiremcnis  docinaent;  a  test  pUn  and  appropnate  confinnation  events;  and  performance,  cost, 
and  schedule  parameters  against  which  progress  can  be  measured  Confinnaiinii  evems  may  be 
operational  tests,  as  well  os  appropriate  trainins  and  oxpenmeniatiDn  eveius  that  are  conducted  in 
the  field,  such  as  the  Division  Capstone  Exercise  slated  €ar  2QFY01.  These  events  must 
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danonsiraie  in  ihe  openuioaal  environment  th&t  functiomUity  is  migrating  toward  objective 
requirements. 

Tbe  Army  will  work  with  Acquisition  Program  Integration  (API)  and  OASD{C3D  to  re- 
baseline  die  five  ACAT I  programs  into  separate  program  blocks  diar  better  reptesem  the  cuzienr 
efSaris  ami  synchronize  requirements  schedules  with  ABCS  software  releases.  The  new 
Acquismon  Program  Basdines  will  include  the  Fiisr  Digitized  Division  and  the  First  Digitized 

Coxps  as  common  schedule  milestones  where  an  imegratedprodua  will  be  deliveted.  For  each 

program  blodc  meeting  Selected  Acquisition  Report  (SAR)  tenmnation  criteria,  the  Army  wiU 
prepare  a  final  SAIL  Should  any  of  the  programs  remain  AC  AT  1  programs,  API  will  work  with 
die  other  OSO  agencies  lo  ensure  DABS  reporting  requirtmcnis  capture  infonnation  of  interest  to 
the  DaE  that  complements  mfonnaxion  m  ^e  Quancrly  Integrated  Summaiy. 

The  Army  will  woric  with  Defense  Procurement  (DP)  to  implement  modular  contracimg  on 
the  five  programs.  The  Army,  including  elements  outside  the  Anny  Acquisirion  Executive 
(AAE)  chain,  will  manay  change  to  achieve  this  common  goal:  requirements  for  each 
contraamcrcixicm  remain  siabk  once  that  inciemcnr  is  underway.  Stable  contract  zequiremencs, 
stated  in  terms  of  the  pctfonnancc  or  ftmchonalny  required  in  deliverable  software,  can  coexist  * 
with  die  imeraenve  software  dcvclopmenr  process  in  vdiich  the  development  contractor  makes 
use  ofongomg  user  feedback.  Toward  this  end,  lequnemcnis  must  be  effectively  coordinated 
before  contract  award,  and  user  feedback  should  naturally  be  withm  the  boundaries  set  by  the 

contraa  requirements.  After  one  modular  contracting  cycle  has  taken  place,  the  Army  and  DP 

will  assess  and  report  on  improvements  in  requireroents  stabiliiy  attained,  the  utili^  of  the 
modular  cantracting  approach  in  achieving  cost  control  and  contractor  accountability,  3>w/f 
adaptations  made  in  the  Anny  requiremenis  process. 

The  C3ISR  &  Space  Systems  DIPT  will  oversee  the  ATCCS/ABCS  sysiem-of-systems.  An 
OSD/Army  IPT  was  an  efGsctive  means  for  developing  the  new  approach  and  should  now 
develop  a  plan  for  n^lemexurng  the  approach,  including  die  recotnmendationj  briefed  on  October 
29,  in  accordance  with  die  guidance  in  this  memorandum.  The  IPT  will  xepon  in  12-18 
the  progress  made  in  implemcnimg  the  plan,  including  progress  on  initiatives  for  broader 
^U<^on  in  DoD  ^mal  development,  sysicm-of-systcms  micgrmtion  and  oversight,  and 
mclusion  into  DoD  acquisition  policy  as  appropriate. 


Attachment 

Distribution: 

PDUSD(AAT) 
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USD(C3 
ASD(C3I) 
aSO(S&TRJ 
army  A£ 

NAVVAE 
AIR  FORCE  AE 
DDR&E 
DUSD(AR) 

DUSD(AT) 

DUSDa&CP) 

DUSD(1A&IJ 

DUSDCD 

DIR,  program  analysis  &  evaluation 
OOT&E 

DXR,  strategic  &  TACTICAL  SYSTEMS 

DIR,  TEST,  SYSTEMS  ENGINEERING  £  EVALUATION 

DIR,  ACQUISITION  PROGRAM  INTEGRATION 

DIR,  DEFENSE  PROCUREMENT 

DDL  SPECIAL  PROGRAMS 

BMDO/PO 

ARMY  PEO  CIS 


Appendix  C.  Remote  Workstation  Users’ 

Observations 


Table  C-1.  Overall  Assessment  of  Operational  Effectiveness  and  Suitability 


User  Comments  Addressing 

Svstem  Limitations 

Fort 

Gordon 

Fort 

Campbell 

Fort 

Bliss 

Fort 

Hood 

Fort 

Lewis 

Nationa 

1 

Guard^^ 

The  system  was  able  to  maintain 
around-the-clock  operations 

No 

No 

No 

No 

No 

No 

Training  was  adequate  and/or  frequent 

No 

No 

No 

No 

No 

No 

Certain  environmental  conditions 
affected  hardware  operations 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

System  operations  were  overly 
dependent  on  contractor  support 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

The  system  is  multi-service 
interoperable 

No 

No 

No 

No 

No 

No 

Table  C-2.  Operational  Suitability 


User  Comments  Addressing 

Svstem  Limitations 

Fort 

Gordon 

Fort 

Campbell 

Fort 

Bliss 

Fort 

Hood 

Fort 

Lewis 

Field  Support 
Contractors 

National 

Guard 

Mobility 

Impairs  unit  movements 

Yes 

Yes 

Yes 

Yes 

Too  large 

Yes 

Yes 

Yes 

Yes 

Setup  and  teardown  times  vary 

Ease  of  Operation 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Familiar  with  UNIX  operating 
system 

No 

No 

No 

No 

Consistent  menu  formats 

No 

No 

Overall  system  is  difficult  to  operate 

Personnel 

Yes 

Yes 

Yes 

Yes 

Understaffed 

Yes 

Yes 

Yes 

Yes 

Yes 

Shortage  of  trained  personnel 

Training 

Yes 

Yes 

Yes 

Yes 

Mostly  on-the-job-training 

Yes 

Yes 

Yes 

Yes 

Available  time  to  train 

No 

No 

No 

Need  leadership  training 

Yes 

Yes 

Yes 

Perishable  operator  skills 

Yes 

Yes 

Yes 

Yes 

Sufficient  training  for  analytical  skills 

Contractor  Support 

No 

No 

No 

Operators  can  troubleshoot 

No 

No 

No 

Response/repair  times  vary 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

National  Guard  installations  visited  included  Camp  Mabry,  Camp  Murray,  and  Kent  Armory. 
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Table  C-3.  Operational  Effectiveness 


User  Comments  Addressing 

System  Limitations 

Fort 

Gordon 

Fort 

Campbell 

Fort 

Bliss 

Fort 

Hood 

Fort 

Lewis 

Field  Support 
Contractors 

National 

Guard 

Hardware 

Adequate  processing  speed 

No 

No 

No 

No 

No 

Slow  repair  times 

Adequate  repairs 

Yes 

Yes 

No 

Can  handle  advanced  software 

No 

No 

Adequate  peripheral  support 

Interoperability 

No 

No 

No 

No 

No 

No 

Software  versions  are  interoperable 

No 

No 

Lacks  communication  with 
other  systems 

Yes 

Yes 

Yes 

System  can  operate  in  a 
stand-alone  mode 

Yes 

Yes 

Yes 

Can  share  graphics 

No 

No 

No 

Received  incorrect  threat  information 

Yes 

Yes 

Message  Rates 


Message  traffic  not  getting  through 

Yes 

Incorrect  text  recognition 

Yes 

Yes 

Yes  Yes 

Yes 

Incorrect  data  correlation 

Message  overload  upon  connection 

Software 

Yes 

Yes 

Yes 

Yes 

Unstable  (crashing/freezing) 

Yes 

Yes 

Yes 

Yes 

Yes 

Version  6  lacks  functionality 

Yes 

Excessive  patches 

Yes 

Yes 

Yes 

Yes 

Too  much  code 

Yes 

Yes 

Fielding 

Delivery  priorities  need  to  be 

reassessed 

Yes 

Disruptive  software  releases 

Yes 

Yes 

Adequate  operational  testing 

Need  better  communication  on 

No 

No 

future  software  versions 

Yes 

Yes 

Usage 

Required  for  everyday  mission 

No 

No 

Used  for  exercises/training  only 

Yes 

Yes 

Yes 
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Table  C-4.  Other  Operational  Issues 


User  Comments  Addressing 

System  Limitations 

Fort 

Gordon 

Fort 

Campbell 

Fort 

Bliss 

Fort 

Hood 

Fort 

Lewis 

Field  Support 
Contractors 

National 

Guard 

Commander  Perceptions 

Because  system  is  deployed,  soldiers 
are  required  to  make  the 
system  work 

Yes 

Yes 

Yes 

Yes 

Yes 

Lack  of  trust 

Yes 

Yes 

Yes 

Yes 

Yes 

Unrealistic  expectations 

Yes 

Yes 

Yes 

Yes 

Yes 

Forced  to  use  the  system 

Desired  Improvements 

Yes 

User  friendliness 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Commercial  off-the-shelf  products 

Yes 

Yes 

Yes 

Increased/standardized  functionality 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Multi-Service  interoperability 

Yes 

Yes 

Yes 

Yes 

Mobile  and  smaller 

Yes 

Yes 

Yes 

Yes 

Yes 

Actions  To  Compensate  For  Lost 
Capability 

Backup  map  boards 

Yes 

Yes 

Yes 

Yes 

Yes 

PowerPoint 

Yes 

Yes 

Yes 

Yes 

Yes 

Use  of  other  systems 

Yes 

Yes 

Yes 

Yes 

Yes 

Radio/telephone 

Yes 

Yes 

Websites 

Yes 

Yes 

Yes 

Yes 

Yes 

Segregated  server  for  booting 
Constant  rebooting 

Yes 

Yes 

Yes 

Supplemental  actions  are 
standard  procedures 

Yes 

Yes 

Yes 
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Appendix  D.  Assessments  of  Operational  Test 

Results  for  Measuring  Remote 
Workstation  Effectiveness 


Table  D-1.  Critical  Operational  Issues  and  Criteria  1 
Does  the  ASAS  Satisfy  the  Commander’s  Intelligence  and  Targeting  Support  Requirements? 


Criteria  Met 

Criteria  Partially  Met 

Criteria  Not  Met 

Criteria 

Deferred 

Imagery  products 

Temnlates 

Target  definitions 

Correlate 

Intelligence  messages  and 

Alerts 

[Collection]  asset  management 

Nodal  analysis 

graphics 

Answer  85  percent  of 

Develop  special  intelligence 

Damage 

Large  screen  graphics 

priority  intelligence 

requirements 

assessment 

Overlays 

requirements 

Intelligence  synchronization 

All- source 

Historical  tracking 

Red  course  of  action 

matrix/collection  plan 

interoperability 

Situation  messages/graphics 

Criteria  to  identify  high 

Track  90  percent  of 

Receive  weather 

Generate  target  intelligence 

priority/interest 

[collection]  assets 

data 

data 

information 

Match  [collection] 

(2  minutes) 

Nominate  targets 

assets/requirements 

Targeting  messages 

Targeting  criteria 

Tasking  messages 

Queries  (less  than  1  minute) 

Nominate  70  percent  of 

Install  80  percent  of  databases 

high  priority  targets 

No  information  loss  due  to 

Track  [collection] 

power  failure 

requirements 

Collection  messages 

1  Relevant  common  picture  1 

Auto  forward  messages 

1  ‘ _ 

Table  D-2.  Critical  Operational  Issues  and  Criteria  2 
Can  the  ASAS  Establish  and  Maintain  Interfaces  to  Provide  Required  Information  Exchanges? 


Criteria  Met 

Criteria  Partially  Met 

Criteria  Not  Met 

Criteria  Deferred 

Messages/graphics/imagery 
Backward  compatible 

External  imagery 

Home  page 

n/a 

n/a  J 

Analysis  and  Control  Element 
processes  messages 

Analysis  and  Control  Element’s 
levels  of  classification 

Interfaces  with  command,  control, 
communications  and 
intelligence 

Systems  Analysis  and  Control 
Element  interface  with  Army 
Battlefield  Command  and 

Control  System 

Interfaces  with  DoD 

■ 

■ 

□  Denotes  Disagreement  Between 
U.S.  Army  and  Audit  Findings 


■  Denotes  Significant  System 
■  ■■■  Attributes  Not  Tested 
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Appendix  E.  Assessments  of  Operational  Test 

Results  for  Measuring  Remote 
Workstation  Suitability 


Table  E-1  Critical  Operational  Issues  and  Criteria  3 
Can  ASAS  deploy  and  operate  on  the  battlefield? 


Criteria  Met 

Criteria 

Partially 

Met 

Criteria 

Not  Met 

Criteria  Deferred 

Criteria  Not  Evaluated 

Set-up  (45  min.), 
tear  down 
(30  min.) 

Seeurity  Level 

Time  to  repair 
(1  hr) 

Meets  reliability  & 
Maintainability 
standards  for 
government/ 
eommereial  off 
the-shelf  systems 

N/A 

Add/delete 

workstations 

Extended  operations  Analysis  and 
Control  Element  (ACE) 

Jump  operations 

ACE  resynehronization 

ACE  split-based  operations 
Signature  suppression 

Nuelear,  biologieal  and  ehemieal 
eonditions 

Deeontamination 

Mission-oriented  proteetive 
posture  IV 

Battlefield  eonditions 

Destroy  elassified 

Restore  system  funetion 
(min.  3  hr,  full  12  hr) 

80  pereent  availability  rate 
Eleetromagnetie  environmental 
effeet 

Information  operations 

The  ASAS  must  be 
deployable  on  C- 130  and 
larger  aireraft 

The  ASAS  must  be  rail 
transportable 

The  ASAS  must  be 
transportable  externally 
by  CH-47 

The  ASAS  will  meet  U.S. 
and  NATO  highway 
legal  limits 

The  ASAS  must  be  marine 
transportable  on  LACV 

30  or  larger  vessels 

Table  E-2.  Critical  Operational  Issues  and  Criteria  4 
Can  the  ASAS  operators/analysts/maintainers  achieve  a  level  of  system  training  readiness  during 
peacetime  that  provides  a  wartime  capability  for  sustained  combat? 


Criteria  Met 

Criteria  Partially 
Met 

Criteria  Not  Met 

Criteria  Deferred 

Criteria  Not 
Evaluated 

Logisties  support 

Software  maturity 

System  stability 

System  robustness 

Subsystem  interoperability 

1  Soldier  sustainment  | 

n/a 

Leave  behind  training 
program 

On-line  help,  CD  ROM 

n/a 

Denotes  Disagreement  Between  Denotes  Significant  System 

U.S.  Army  and  Audit  Findings  Attributes  Not  Tested 
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Appendix  F.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Under  Secretary  of  Defense  (Comptroller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Under  Secretary  of  Defense  for  Intelligence 

Assistant  Secretary  of  Defense  for  Networks  and  Information  Integration/DoD  Chief 
Information  Officer 

Director,  Operational  Test  and  Evaluation 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 

Assistant  Secretary  of  the  Army  (Acquisition,  Logistics  and  Technology) 

Program  Executive  Officer,  Command,  Control  and  Communications  Tactical 
Project  Manager,  Intelligence  and  Effects 

System  Manager  for  All  Source  Analysis  System,  Training  and  Doctrine  Command 
Commander,  Army  Test  and  Evaluation  Command 
Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Auditor  General,  Department  of  the  Air  Force 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Governmental  Affairs 

Senate  Select  Committee  on  Intelligence 

House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency  and  Financial  Management,  Committee 
on  Government  Reform 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Information  Policy,  and  Intergovernmental 
Relations,  and  the  Census,  Committee  on  Government  Reform 
House  Permanent  Select  Committee  on  Intelligence 
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Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  Comments 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

SOOO  DCFEMSe  F^ENTAGON 
WASHII^TON.  DC  ^0301-^000 


SEP  1  e  im 


MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  [NSPECTOR  GENERAL 

SUBJECT:  DoD  IG  Draft  Report,  "Acquisition  Manag^sment  of  the  Army's  Ail  Source 
Analysis  System  (ASAS)  ”  (Project  No,  D2002AU0157) 


iTiis  is  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 
leyponse  to  the  DoD  IG  Draft  Report,  “Report  on  the  Acquisdtion  Management  of  the 
Army's  All  Source  Analysis  System  (Project  No.  D2002ALp0157), 

We  have  reviewed  and  noc-concur  with  recommeEdation  1  .a,  and  concur  with 
omiinenc  on  lecomtnCTnktioiis  Lb.  and  2,  of  the  subject  report,  see  enclosure  for  detailed 
comments. 

NancyU  Spruill 
Director*  Acquisition  Resources 
and  Analysis 


ALLachment: 
As  staled 
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IG  DRAFT  REPORT  -  DATED  JUNE  30, 2003 
PROJECT  NO.  D2002AL-0157 

ACOUlSmON  MANAGEMENT  OF  THE  ARMY’S  ALL  SOURCE 
ANALYSIS  SYSTEM 

DEPARTMENT  OF  DEFENSE  RESPONSE  TO  THE 
RECOMMENDATIONS 


RECOMMENDATION  la.  Recommend  the  USIHAT&L)  redfisignate  All  Source 
Analysis  System  as  an  Acquisition  Category  1  Majw  Defense  Acquisition  Program 
in  accordance  with  DoD  guidance. 

DoD  RESPONSE:  Nonconcur  with  comioent:  The  DoD  Major  Defense 
Acquisition  Program  (MDAP)  list  will  be  updated  prior  to  the  end  of  calendar  year 
2003.  The  ASAS  will  be  evaluated  prior  to  the  end  of  CY2003  in  accordance  with 
the  dollar  thresholds  of  Title  10,  Section  2430,  United  States  Code  for  MDAP,  to 
determine  the  appropriate  MDAP  level. 


PFrOMlVP^NDATlON  lb:  Recommend  the  USD(AT&L)  assume  responsibility 


for  providing  DoD  acquisition  management  ovasight  for  the  All  Source  Analysis 


System  Block  11  acquisition. 


DqD  RESPONSE:  Concur  with  comment:  If  AS  AS  meets  the  statutory  dollar 
thresholds  for  an  MDAP*  it  will  be  designated  an  Acquisition  Category  (ACAT)  I 
program  with  OUSD(AT&L)  acquisition  management  oversight 

RECOMMENDATION  2:  Recommend  that  die  USD(AT&L)  in  coordination  with 
the  Under  Secretary  of  Defense  (Intelligence),  raise  the  deployment  threshold  for 
the  All  Source  Analysis  System  operational  tests  from  “migrating  towards 
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objective  lequiremenls”  to  satisfying  benchm^erks  for  critical  operationaJ  issues 
and  crileiia. 

DoD  RESPQNSE:  Conciir  with  ccaaitiimt;  If  ASAS 15  designated  an  ACAT I 
progrButy  die  USD{AT&L)  in  cooidination  with  the  USD(I)  and  EJOT&E  will 
issue  wprfai'wt  pEOgFam  gakSancey  n>  include  lerisions  to  Ibe  Acquisition  Program 
fiasdine  (APB),  Acquisition  Strategy,  and  Test  and  EvaloatiDn  Master  Plan 
(TEMP). 
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Director,  Operational  Test  and  Evaluation 
Comments 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
1700  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301*1700 


AUG  1  9  2009 


MEMORANDUM  FOR  INSPECTOR  GENERAL  OF  THE  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Report  on  the  Acquisition  Management  of  the  Ann/s  All  Source  Analysis  System 
(ASAS),  dated  June  30, 2003 


Wc  have  reviewed  the  subject  document  and  are  unable  to  respond  to  the  recommendations 
at  this  time  due  to  discrepancies  in  the  report's  analysis  of  the  ASAS  test  program.  The  office 
provided  a  number  of  comments  on  the  draft  of  this  report.  These  are  significant  concerns  and 
1  ask  that  the  audit  team  contact  my  staff  to  discuss  them. 

Wc  have  closely  monitored  the  intermediate  release  of  all  the  ASAS  components  and  have 
approved  their  operational  test  plans.  Wc  have  worked  closely  with  the  Army  Test  and 
Evaluation  Command  on  their  assessments  and  find  no  misrepresentation  of  test  results  or 
performance  assessments.  Operational  tests  were  adequate  representations  of  an  operational 
environment. 

DOT&E  will  provide  an  assessment  of  operational  effectiveness,  survivability,  and 
suitability,  as  well  as  test  adequacy  of  the  ASAS  Block  n  family  of  systems  following  the 
conclusion  of  the  Block  n  Initial  Operational  Test  and  Evaluation  event  scheduled  for  October 
2004. 

The  point  of  contact  for  this  action  is  Ms.  Susan  J.  Wright,  (703)  681-1440  ext  1 15, 
susan.wright@osd.mil. 


fmas  P^hristie 
/qqXoiJ/ 


J.}uUU^ 


Department  of  the  Army  Comments 


DEPARTMENT  OF  TUB  ARMY 

PftDGflAM  EXECUTIVE  OFFICE 

coMyANOp  cohthol,  and  communications  tactical 

FONT  UPNMOUTH  NEW  JERSEY  07/03-5401 
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SFAE-C3T-OPS 


MEMORANDUM  FOR  INSPECTOR  GENERAL^  DEPARTMEN  I’  OF  DEFENSE,  400 
ARMY  NAVY  DRIVE,  ARLINGTON,  VA  22202-4704 

SUBJECT:  Report  on  the  Acquisition  Maniigcmcm  of  the  Army's  All  Source  Analysis  System 
(Project  No.  D2002AL-0 1 57) 

1 .  Program  Executive  Office  Command,  Control  and  Commimications  Tactical  (PEO  C3T)  has 
reviewed  the  subject  Inspector  General  Report  and  submits  a  concurrence.  PEO  C3T  is  working 
with  the  Program  Management  OlTice  to  re-establish  baseline  integrity  NI^T  1 5  October  2003 
by  ensuring  that  cost,  schedule  and  peiformance  parameters  measure  progress^  determine 
cfTcetiveness  and  efficiency  indexes,  and  project  program  result*?. 

2r  If  you  have  any  questions  on  this  matter,  please  direct  them  to  Mr.  Edw  ard  Karol,  DSN: 
9S7-17S^or  COML:  732-427-1789 


IN  C.  PERR.^PATO 


Deputy  Program  Execulive  Officer  for  C3T 
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Team  Members 


The  Acquisition  Management  Directorate,  Office  of  the  Deputy  Inspector  General 
for  Auditing  of  the  Department  of  Defense  prepared  this  report.  Personnel  of  the 
Office  of  the  Inspector  General  of  the  Department  of  Defense  who  contributed  to 
the  report  are  listed  below. 

Charles  M.  Santoni 
David  M.  Wyte 
Stephen  J.  Bressi 
George  A.  Leighton 
MAJ  William  C.  Busch 
Robert  R.  Johnson 
John  R.  Huddleston 
Leann  A.  Alwood 
Jacqueline  N.  Pugh 


